Preprint ^from:^  PROCEEDINGS  OF  THE 
HUMAN  FACTORS  SOCIETY 
28th  Annual  Meeting  1984 
San  Antonio,  "fexas  Oct,  1984 


SUBJECTIVE  WORKLOAD  ASSESSMENT  IN  A  SPATIAL  MEMORY  TASK 
F.  Thomas  Eggemeier 


Twelve  subjects  performed  a  spatial  short-term  memory  task  under  several  levels  of 
difficulty  and  rated  the  workload  associated  with  each  using  the  Subjective  Workload 
Assessment  Technique  (SWAT).  SWAT  ratings  proved  sensitive  to  two  of  the  three  diffi¬ 
culty  manipulations  in  the  memory  task,  and  demonstrated  greater  sensitivity  in  this 
respect  than  either  of  two  primary  task  measures  that  were  employed.  The  results  extend 
the  applicability  of  SWAT  to  the  type  of  spatial  memory  task  used  and,  therefore,  provide 
further  support  for  the  general  applicability  of  SWAT  as  a  workload  measurement 
technique. 


INTRODUCTION 

A  wide  variety  of  assessment  techniques 
have  been  proposed  as  measures  of  operator 
workload  (e.g.,  Willi ges  and  Wierwille, 
1979).  These  techniques  vary  in  their  capa¬ 
bility  to  meet  the  objectives  and  practical 
constraints  of  different  applications.  Con¬ 
sequently,  a  comprehensive  approach  to  work¬ 
load  assessment  will  require  the  use  of 
several  classes  of  assessment  techniques 
(e.g.,  Eggemeier,  1984;  Shi ngl edecker,  1983). 

Subjective  measurement  techniques  are 
capable  of  satisfying  a  number  of  practical 
constraints  (e.g.,  ease  of  implementation, 
lack  of  intrusion  on  operator  performance) 
which  must  be  met  in  many  applications  and, 
therefore,  represent  a  viable  candidate  for 
inclusion  in  a  comprehensive  workload  assess¬ 
ment  methodology.  Because  of  their  practical 
utility,  subjective  techniques  have  been  fre¬ 
quently  used  to  measure  operator  load  in  a 
variety  of  laboratory  and  applications  envi¬ 
ronments  (Moray,  1982;  Williges  and  Wierwille, 
1979). 

Despite  the  widespread  use  of  subjective 
measures,  the  rating  scale  literature  has  been 
characterized  by  individual  development  of 
techniques  for  particular  applications.  With 
few  exceptions  (e.g.,  Wierwille  and  Casali, 
1983),  rating  scales  have  not  been  evaluated 
with  respect  to  their  sensitivity  to  workload 
variations  in  a  variety  of  tasks.  Therefore, 
there  is  little  evidence  in  the  current  lit¬ 
erature  of  a  workload  rating  technique  that 
can  be  recommended  for  generalized  use  as  part 
of  an  overall  workload  assessment 
methodology.  In  order  to  provide  a  workload 


rating  scale  with  the  potential  for  general 
applicability,  the  Subjective  Workload  Assess¬ 
ment  Technique  (SWAT)  was  developed  (Reid, 
Shingledecker,  and  Eggemeier,  1981a;  Reid, 
Shingledecker,  Nygren,  and  Eggemeier,  1981b). 

In  SWAT,  subjective  workload  is  assumed 
to  be  determined  by  loading  on  three  major 
dimensions:  (1)  time,  (2)  mental  effort,  and 
(3)  stress.  Time  load  represents  the  percen¬ 
tage  of  time  that  the  operator  is  occupied 
with  information  processing  requirements, 
mental  effort  refers  to  the  degree  of  concen¬ 
tration  and  attention  required  during  perform¬ 
ance,  and  stress  load  represents  any 
additional  factors  that  lead  to  confusion  or 
anxiety  during  performance.  Each  dimension  is 
represented  by  a  3-point  rating  scale  with 
verbal  descriptors  that  outline  levels  on  each 
dimension.  SWAT  represents  an  application  of 
conjoint  measurement  and  scaling  (e.g.,  Krantz 
and  Tversky,  1971;  Nygren,  1982)  which  permit 
ratings  on  the  three  dimensions  to  be  combined 
into  one  overall  interval  scale  of  workload. 

Application  of  SWAT  is  a  two-step 
process.  During  an  initial  scale  development 
phase,  subjects  rank  order  the  subjective 
workload  associated  with  all  possible  combina¬ 
tions  of  time,  effort,  and  stress  load.  This 
ordering  information  is  used  to  determine  the 
rule  for  combining  the  three  dimensions  into 
the  overall  interval  scale  that  results  from 
the  procedure.  During  the  second  or  event- 
scoring  phase,  subjects  use  each  of  the  three 
individual  scales  to  rate  the  levels  of  load¬ 
ing  associated  with  performance  of  a  particu¬ 
lar  task.  The  combination  of  ratings  on  the 
three  scales  is  then  used  to  specify  a  value 
on  the  overall  interval  scale  that  was  derived 


Xli..  t--  sF  ' 


i ...  ia  J3  i 


MjppsaraS.  fm  pah Bs  ssfess©!  I 

fcraScs  Uslnsltsdl  ? 


19950104  084 


f 


during  scale  development.  More  detailed  dis¬ 
cussions  of  the  SWAT  procedure  can  be  found  in  ' 
Reid  et  al .  (1981a,  b). 

An  important  aspect  of  the  SWAT 
development  program  has  been  determining  the 
sensitivity  of  the  technique  to  workload 
variations  in  a  number  of  different  types  of 
tasks.  Previous  investigations  have  demon¬ 
strated  that  SWAT  is  sensitive  to  variations 
in  the  difficulty  of  several  laboratory  tasks, 
including  critical  tracking  and  simulated  air¬ 
crew  radio  communications  (Reid  et  al., 
1981a),  display  monitoring  (Notestine,  1983), 
and  verbal  short-term  memory  (Eggemeier, 
Crabtree,  Zingg,  Reid,  and  Shingledecker, 
1982).  SWAT  ratings  have  also  proven  sensi¬ 
tive  to  variations  in  several  forms  of  task 
loading  in  high  fidelity  flight  simulations, 
such  as  the  number  of  opponents  in  aerial  com¬ 
bat  (Reid,  Eggemeier,  and  Shingledecker,  1984) 
and  the  presence  or  absence  of  threats  to  an 
aircraft  (Skelly,  Reid,  and  Wilson,  1983). 

The  purpose  of  the  present  study  was  to 
further  explore  the  general  applicability  of 
the  SWAT  technique  by  examining  its  sensitiv¬ 
ity  to  a  number  of  difficulty  manipulations  in 
a  spatial  short-term  memory  task.  The 
requirement  to  process  and  remember  spatial 
information  is  a  central  component  of  operator 
activity  in  many  complex  situations  (e.g.,  air 
traffic  control,  piloting  an  aircraft).  Also, 
some  current  descriptions  (e.g.,  Wickens, 
1984)  of  limits  within  the  human  system  sug¬ 
gest  that  spatial  information  processing  draws 
on  capacities/resources  that  are  separate  from 
those  used  in  processing  verbal  materials. 
Therefore,  it  was  considered  important  to 
determine  the  sensitivity  of  SWAT  loading  to 
variations  in  a  task  which  involved  spatial 
processing.  The  task  used  in  this  experiment 
was  a  modified  version  of  a  pattern  recog¬ 
nition  task  that  was  originally  developed  by 
Chiles,  Al  lui si ,  and  Adams  (1968).  The  cur¬ 
rent  task  required  that  subjects  retain  a 
histogram  pattern  over  a  short  retention 
interval  and  indicate  whether  a  subsequently 
presented  histogram  matched  the  original  or 
"target"  pattern.  Task  difficulty  was  manipu¬ 
lated  by  varying  the  complexity  and  spatial 
orientation  of  the  target  histogram  and  by 
varying  the  length  of  the  retention  interval. 

METHOD 

Subjects 

Subjects  (Ss)  were  five  male  and  seven 
female  introductory  psychology  students  at 
Wright  State  University.  _Ss  received  extra 
course  credit  for  their  participation. 

Apparatus 

Memory  stimuli  were  presented  on  a 
12-inch  black  and  white  video  monitor  that  was 


controlled  by  a  Commodore  VIC-20  microcom¬ 
puter.  _Ss  were  seated  approximately  50  cm  in 
front  of- the  monitor,  and  responded  by  press¬ 
ing  one  of  two  buttons  on  a  keypad  that  was 
placed  a  comfortable  distance  in  front  of  the 
S/s  preferred  hand. 

Procedure 

At  the  beginning  of  the  session,  _Ss  com¬ 
pleted  the  scale  development  phase  of  the  SWAT 
procedure.  _Ss  were  given  a  deck  of  27  cards, 
each  of  which"  contained  statements  that  repre¬ 
sented  one  of  the  possible  combinations  of  the 
time,  mental  effort,  and  stress  dimensions 
from  the  3-point  SWAT  scales.  _Ss  sorted  the 
cards  so  that  the  27  combinations  were  rank- 
ordered  to  reflect  the  degree  of  subjective 
workload  imposed  by  each.  A  Kendall  coeffi¬ 
cient  of  concordance  (W  =  .78,  p  <  .01)  demon¬ 
strated  significant  agreement  among  Ss  with 
respect  to  rank  orderings  of  the  27  combina¬ 
tions.  Therefore,  the  card  sort  data  from  all 
Ss  was  combined  during  the  SWAT  scaling  pro¬ 
cedure  to  form  one  overall  interval  scale  of 
workload  with  a  range  of  0  to  100.  Values 
from  this  scale  were  then  used  as  the  measure 
of  subjective  load  in  subsequent  analyses. 


Each  trial  in  the  spatial  memory  task 
began  with  presentation  of  the  target 
histogram  for  a  3-second  period  on  the  video 
monitor.  The  target  histogram  was  then 
replaced  by  a  three-digit  number  which 
remained  on  the  screen  for  the  duration  of  the 
retention  interval.  _Ss  counted  backwards 
successively  by  three's  from  this  number  in 
order  to  prevent  any  verbal  rehearsal  of  the 
target  pattern  information.  The  counting  was 
done  aloud  to  ensure  that  S_s  performed  the 
subtraction  task  throughout  the  retention 
interval.  At  the  completion  of  the  retention 
interval,  the  number  was  replaced  by  a  second 
or  comparison  histogram.  _Ss  indicated  whether 
this  histogram  was  the  same  or  different  than 
the  target  pattern  by  pressing  the  appropriate 
button  on  the  keypad  as  quickly  as  possible. 
Length  and  configuration  of  the  bars  in  the 
target  histogram  were  chosen  at  random  by  the 
program,  as  was  the  choice  of  whether  the 
second  histogram  would  be  the  same  or  differ¬ 
ent  than  the  first.  If  it  was  different,  the 
program  again  determined  the  length  and  con¬ 
figuration  of  the  bars  at  random.  The  com¬ 
puter  also  recorded  _S_s '  responses  and  reaction 
times. 
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difficulty  was  manipulated  by 
complexity  of  the  target  histogram 
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or  comparison  histogram  was  always  presented 
in  the  upright  position.  Retention  interval 
was  a  between-Ss  variable,  with  six  _Ss 
at  each  interval.  Within 
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both  retention  interval  groups,  each  _S 
performed  a  block  of  ten  trials  with  each' 
possible  combination  of  the  two  histogram 
complexity  levels  and  three  orientations  for  a 
total  of  60  trials.  The  order  of  these 
combinations  was  counterbalanced  across  the  _Ss 
in  each  retention  interval  group.  After  each 
block  of  10  trials,  Ss  rated  the  time,  mental 
effort,  and  stress  Toad  imposed  by  the  com¬ 
bination  of  complexity,  orientation,  and 
retention  interval  represented  in  the  block. 

Prior  to  actual  data  collection,  Ss 
received  practice  on  the  spatial  memory  task 
and  on  performing  SWAT  ratings.  After  a  short 
demonstration  of  the  task,  _Ss  performed  blocks 
of  ten  practice  trials  under  each  of  the  fol¬ 
lowing  conditions:  two  bars  at  0-degree 

orientation,  four  bars  at  each  orientation, 
and  six  bars  at  the  180-degree  orientation 
level.  The  retention  interval  for  all  prac¬ 
tice  blocks  was  24  seconds.  _Ss  practiced  SWAT 
ratings  during  the  four-bar  conditions,  but 
not  in  the  two-  or  six-bar  conditions  since 
these  would  be  rated  during  actual  testing. 

RESULTS 


The  SWAT  rating  data  were  analyzed  using 
a  three-factor  analysis  of  variance  (ANOVA). 
Two  levels  of  retention  interval  (16, 
32  seconds),  two  levels  of  histogram  complex¬ 
ity  (two,  six  bars),  and  three  levels  of 
orientation  (0,  90,  180  degrees)  were  included 
in  the  ANOVA.  The  ANOVA  indicated  that  both 
the  main  effect  of  retention  interval 
[F(  1 , 10)  =  7.57,  p  <  .05]  and  histogram  com¬ 
plexity  [F(l ,10)  =  16.69,  p  <  .01]  were  sta¬ 
tistically  significant.  However,  neither  the 
main  effect  of  orientation  [F ( 2 ,20)  =  1.81, 
p  >  .10]  nor  any  of  the  interactions  proved  to 
be  significant.  Figure  1  shows  mean  SWAT 
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Figure  1.  Mean  SWAT  Ratings  as  a 
Function  of  Histogram 
Complexity  and  Retention 
Interval 


ratings  as  a  function  of  both  the  significant 
retention  interval  and  histogram  complexity 
factors.  As  is  clear  from  Figure  1,  SWAT 
ratings  increased  with  increases  in  both 
levels  of  complexity  and  retention  interval. 

Both  memory  task  errors  and  reaction 
times  were  analyzed  with  individual  2x2x3 
ANOVAs  that  were  comparable  to  the  ANOVA  per¬ 
formed  on  the  SWAT  rating  data.  The  ANOVA 
performed  on  the  error  data  failed  to  demon¬ 
strate  any  significant  main  effects  or  inter¬ 
actions,  and  so  no  further  analyses  were 
performed  on  these  data.  The  reaction  time 
ANOVA  yielded  a  significant  main  effect  of 
histogram  complexity  [F(l,10)  =  12.69,  p  < 

.01],  but  failed  to  demonstrate  significant 
main  effects  of  retention  interval  [F(l,10)  = 
0.36,  p  >  .25]  or  orientation  [F { 2 , 20 )  =  3.20, 
p  <  .10].  None  of  the  interactions  proved  to 
be  statistically  reliable.  Figure  2  shows 
mean  reaction  time  as  a  function  of  histogram 
complexity  and  retention  interval  and  provides 
a  basis  to  compare  the  reaction  time  results 
with  the  SWAT  ratings  in  Figure  1.  As  with 
the  SWAT  data,  reaction  time  increased  with 
increases  in  histogram  complexity  and,  par¬ 
ticularly  in  the  six-bar  condition,  showed  a 
tendency  to  increase  as  the  retention  interval 
was  lengthened. 
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Figure  2.  Mean  Reaction  Time  as  a  Function 
of  Histogram  Complexity  and 
Retention  Interval 


DISCUSSION 

The  results  generally  provide  support  for 
the  sensitivity  of  the  SWAT  technique  to 
demand  manipulations  in  the  type  of  spatial 
memory  task  employed  in  this  experiment.  The 
SWAT  procedure  proved  sensitive  to  two  of  the 
three  difficulty  manipulations  that  were  used, 
and  demonstrated  greater  sensitivity  in  this 
regard  than  either  of  the  primary  task 


measures  that  were  employed.  The  results,- 
therefore,  extend  the  applicability  of  SWAT  to 
a  type  of  task  that  had  not  been  previously 
researched  and,  as  such,  provide  further 
support  for  the  general  applicability  of  the 
procedure  as  a  workload  measurement  technique. 

Although  SWAT  failed  to  discriminate 
levels  within  the  orientation  difficulty 
manipulation,  this  was  also  true  of  the  two 
primary  task  measures.  Apparently,  any  addi¬ 
tional  levels  of  load  that  were  imposed  by 
this  manipulation  were  lower  than  those 
associated  with  either  the  complexity  or 
retention  interval  manipulations.  One  poten¬ 
tial  factor  which  could  have  minimized  the 
orientation  manipulation  effect  is  the  fact 
that  while  the  target  histogram  appeared  in 
several  orientations,  the  comparison  histogram 
was  always  presented  in  the  upright  posi¬ 
tion.  Therefore,  subjects  could  have  adopted 
the  strategy  of  mentally  rotating  the  target 
histogram  to  the  upright  position  before 
storing  the  image  for  later  comparison  with 
the  second  histogram.  Some  support  for  the 
feasibility  of  this  strategy  and  for  its 
possible  effect  on  the  reaction  time  measure 
can  be  derived  from  previous  work  on  the  rota¬ 
tion  of  mental  images  (Cooper  and  Shepard, 
1973).  Cooper  and  Shepard  (1973)  demonstrated 
that  increases  in  reaction  time  associated 
with  orientation  differences  between  a  target 
and  comparison  image  could  be  eliminated  by 
providing  advance  information  to  subjects  con¬ 
cerning  the  orientation  of  the  comparison 
figure.  However,  the  information  was  effec¬ 
tive  only  if  it  was  provided  in  time  for 
subjects  to  mentally  rotate  the  target  figure 
prior  to  its  comparison  with  the  second 
image.  Since  subjects  in  the  present  study 
always  had  advance  information  about  the 
orientation  of  the  comparison  histogram,  and 
since  there  was  sufficient  time  to  rotate  the 
target  prior  to  the  comparison  operation,  it 
appears  that  the  suggested  strategy  would  have 
been  feasible  and  would  account  for  the  lack 
of  reaction  time  differences  as  a  function  of 
target  orientation.  Also,  since  the  rotated 
images  would  have  been  identical  to  those 
stored  on  nonrotation  trials,  memory  storage 
and  retrieval  demands  assoicated  with  the 
orientation  variable  would  have  been  mini¬ 
mized.  Apparently,  this  minimal  additional 
loading  was  not  sufficient  to  be  reflected  in 
either  the  error  measure  or  in  the  SWAT 
ratings. 

The  capability  of  SWAT  to  discriminate 
levels  of  loading  imposed  by  the  retention 
interval  variable  is  noteworthy,  since  neither 
the  memory  error  nor  reaction  time  measure 
demonstrated  significant  differences  as  a 
function  of  this  variable.  This  type  of 
result  is  consistent  with  a  major  rationale 
for  use  of  subjective  assessment  techniques  in 
addition  to  primary  task  performance  measures, 
since  the  latter  are  capable  of  discriminating 


overload  from  nonoverload  situations  (e.g., 
Williges  and  Wierwille,  1979)  but  can  be 
insensitive  to  nonoverload  demand 
manipulations.  It  appears  that  the  retention 
interval  manipulation  used  in  this  experiment 
was  within  the  nonoverload  region  of  the 
workload-performance  relationship,  permitting 
subjects  in  the  32-second  group  to  perform  at 
a  level  comparable  to  the  16-second  group. 
However,  this  equivalent  performance  was 
apparently  achieved  at  the  expense  of  greater 
effort  or  capacity  expenditure  which  was 
reflected  in  the  more  sensitive  subjective 
measure.  This  high  level  of  SWAT  sensitivity 
relative  to  primary  task  performance  is 
consistent  with  previous  results  obtained  with 
a  verbal  short-term  memory  task  in  which  SWAT 
discriminated  levels  of  a  stimulus  presenta¬ 
tion  rate  variable  that  did  not  significantly 
affect  memory  performance  (Eggemeier, 
Crabtree,  and  LaPointe,  1983).  Therefore,  the 
pattern  of  results  in  this  and  the  Eggemeier 
et  al .  (1983)  experiments  suggests  that  an 
advantage  of  SWAT  relative  to  primary  task 
measures  is  an  increased  level  of  sensitivity 
to  some  forms  of  task  demand  manipulation. 

It  is  important,  however,  that  the  noted 
patterns  of  sensitivity  not  be  interpreted  as 
indicating  that  SWAT  should  replace  the  use  of 
primary  task  measures  in  assessment  of 
operator  workload.  Primary  task  measures  can 
provide  valuable  information  about  levels  of 
operator  performance  and  permit  identification 
of  overloads  that  result  in  significant 
performance  degradations.  SWAT,  on  the  other 
hand,  can  provide  valuable  information 
regarding  variations  in  loading  and  the 
potential  for  overloads  that  is  not  reflected 
in  primary  task  performance  during  nonoverload 
conditions.  The  noted  differences  in 
sensitivity  and  in  the  type  of  information 
provided  therefore  suggest  the  complementary 
use  of  both  techniques  in  a  comprehensive 
approach  to  operator  workload  assessment. 
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